The Datathon

The project brief

• Create a data solution

• that tackles one or more of the 17 sustainable development goals

• by leveraging one of the six key transitions

  • food systems
  • energy access and affordability
  • digital connectivity
  • education
  • jobs and security
  • climate change, biodiversity loss, and pollution

• and focuses on the SDG localisation enabler

  • place local communities at the centre of development responses
  • enable local advocacy, local action, and local monitoring and reporting

Our project

Aim

• At the beginning: Solve world problems
• Towards the middle: Slap together a half-baked solution in 3 days.

Concordance

Data was all remapped from their raw forms onto a consistent grid.

rasterGrid = raster(ncols = 3600, nrows = 1800,
                    xmn = -180, xmx = 180,
                    ymn = -90, ymx = 90)
baseRaster = terra::rast(rasterGrid)

rawValues = terra::rast(tiffFile)
consistentValues = resample(rawValues, baseRaster, method = "bilinear")

valueDataFrame = as.data.frame(consistentValues, xy = TRUE, na.rm = FALSE) %>% 
    mutate(id = 1:ncell(consistentValues))

Rough Model Details: Power Ratio

Regress per-area power production of existing solar farm locations on a laughably small number of factors (photovoltaic potential, land use, terrain slope).

Using “spatial” “random forest”.

library(caret)

form = power_ratio ~ biomass + slope + photovoltaic_potential + lat + lon

caret::train(
  form,
  method = "ranger",
  ...
)

Rough Model Details: Energy Demand

Demand was modelled using a proxy quantity constructed from night light intensity and population density

import polars as pl

(regressors
  .with_columns(
    log_pop_density = (255 - pl.col('density') + 1).log10(),
    log_nightlight = (pl.col('nightlight') + 1).log10(),
  )
  .with_columns(
    demand = -(pl.col('log_pop_density') + pl.col('log_nightlight')) + (np.log10(256))
  )
  .select('x', 'y', 'demand')
)

Day 1

So none of us had much experience with spatial data.

Most of the day was spent collecting and sourcing data.

Initial focus was on Africa, but we couldn’t find nice shape files or very local data for the region.

Limitations:

• We wanted to find spatial data at a resolution that was better than at country level, and had data for the entire globe.

• Work was done on an AWS EC2 VM instance that had RStudio Server installed.

  • AWS has an image with RStudio Server Pro, but you need to add the AMI, which we couldn’t do.
  • We chose the wrong instance image (Amazon Linux) and spent a day building dependencies from source.
  • We did this again when we resized the attached EBS storage.

The madness continues…

Day 2

• Data was concorded onto a regular grid for analysis.
  • The focus of our solution was changed to the entire world.
  • The grid size was set to 3600 x 1800 (6,480,000 cells!)
  • Initially done in Python, then refactored into a single call in R

• The data was consolidated and models were built
  • So I decided to make a Shiny app to showcase our final solution
  • Used the rhino framework from Appsilon to speed up the process
  • Learned how to showcase 6 million points in leaflet
    • Had to reduced the resolution to a manageable level
    • Used rectangles to showcase areas, instead of individual dots

End of day:

• A simple linear regression model was built as a baseline model
• Models were trained and validated overnight
• Shiny app was built up to a rough POC

Day 3

Learnings

Acknowledgements